Store access token missing by ph · Pull Request #15089 · elastic/beats

ph · 2019-12-12T20:28:55Z

This PR add a few things

- It takes the keystore encryption implementation and create an
  io.Reader and io.Writer for it, we currently work with variable lenght
  blocks and we are not compatible for now with the one in libbeat.

- It implements a few storage mechanism and wrapper:
  - DiskStore: Save a io.reader directly to disk, the content is saved
    in a temporary file and the target is replace in an atomic
    operation.
  - EncryptedDiskStore: Same as Keystore but use the encrypted io.Reader
    and io.Writer.

- A fleet configuration is created in _meta/agent.fleet.yml, the content
  of this file is packed in the binary. When a user enroll the agent,
  we do a backup of the current agent.yml and the content is replaced
  with the agent.fleet.yml

- The enrollment information is saved into a "fleet.yml"

- The managed mode now read the content of the fleet.yml and creates a
  Kibana API client.

- A Separates Fleet Config struct is created.

- Enroll will now ask for confirmation before replacing the user
  configuration.

ref: #14951

This PR add a few things - It takes the keystore encryption implementation and create an io.Reader and io.Writer for it, we currently work with variable lenght blocks and we are not compatible for now with the one in libbeat. - It implements a few storage mechanism and wrapper: - DiskStore: Save a io.reader directly to disk, the content is saved in a temporary file and the target is replace in an atomic operation. - EncryptedDiskStore: Same as Keystore but use the encrypted io.Reader and io.Writer. - A fleet configuration is created in _meta/agent.fleet.yml, the content of this file is packed in the binary. When a user enroll the agent, we do a backup of the current agent.yml and the content is replaced with the agent.fleet.yml - The enrollment information is saved into a "fleet.yml" - The managed mode now read the content of the fleet.yml and creates a Kibana API client. - A Separates Fleet Config struct is created. - Enroll will now ask for confirmation before replacing the user configuration.

elasticmachine · 2019-12-12T20:28:58Z

Pinging @elastic/ingest (Project:fleet)

ph · 2019-12-13T18:48:20Z

@michalpristas

Have you seen this error before CI?

-- FAIL: TestShortRun (0.00s)

    operator_test.go:36: operation-start: chown /home/travis/gopath/src/github.com/elastic/beats/x-pack/agent/pkg/agent/operation/tests/scripts/short--1.0.yml: operation not permitted

=== RUN   TestShortRunInvalid

2019-12-12T20:55:33Z INFO	operation_fetch.go:80	operation 'operation-fetch' downloaded bumblebee. into 

2019-12-12T20:55:33Z INFO	operator.go:199	operation 'operation-verify' skipped for bumblebee.

2019-12-12T20:55:33Z INFO	process.go:72	address assigned to the process '/bin/bumblebee/bumblebee': '127.0.0.1:10000'

--- PASS: TestShortRunInvalid (1.00s)

=== RUN   TestLongRunWithStop

2019-12-12T20:55:34Z INFO	operation_fetch.go:80	operation 'operation-fetch' downloaded long.1.0 into 

2019-12-12T20:55:34Z INFO	operator.go:199	operation 'operation-verify' skipped for long.1.0

2019-12-12T20:55:34Z INFO	operator.go:199	operation 'operation-install' skipped for long.1.0

2019-12-12T20:55:34Z INFO	process.go:72	address assigned to the process '/bin/sh': '127.0.0.1:10000'

--- FAIL: TestLongRunWithStop (0.00s)

    operator_test.go:71: operation-start: failed to start '/bin/sh': fork/exec /bin/sh: operation not permitted

=== RUN   TestLongRunWithCrash

2019-12-12T20:55:34Z INFO	operation_fetch.go:80	operation 'operation-fetch' downloaded long.1.0 into 

2019-12-12T20:55:34Z INFO	operator.go:199	operation 'operation-verify' skipped for long.1.0

2019-12-12T20:55:34Z INFO	operator.go:199	operation 'operation-install' skipped for long.1.0

2019-12-12T20:55:34Z INFO	process.go:72	address assigned to the process '/bin/sh': '127.0.0.1:10000'

--- FAIL: TestLongRunWithCrash (0.01s)

    operator_test.go:112: operation-start: failed to start '/bin/sh': fork/exec /bin/sh: operation not permitted

=== RUN   TestTwoProcesses

2019-12-12T20:55:34Z INFO	operation_fetch.go:80	operation 'operation-fetch' downloaded long.1.0 into 

2019-12-12T20:55:34Z INFO	operator.go:199	operation 'operation-verify' skipped for long.1.0

2019-12-12T20:55:34Z INFO	operator.go:199	operation 'operation-install' skipped for long.1.0

2019-12-12T20:55:34Z INFO	process.go:72	address assigned to the process '/bin/sh': '127.0.0.1:10000'

--- FAIL: TestTwoProcesses (0.00s)

    operator_test.go:160: operation-start: failed to start '/bin/sh': fork/exec /bin/sh: operation not permitted

=== RUN   TestConfigurableRun

2019-12-12T20:55:34Z INFO	operation_fetch.go:80	operation 'operation-fetch' downloaded configurable.1.0 into 

2019-12-12T20:55:34Z INFO	operator.go:199	operation 'operation-verify' skipped for configurable.1.0

2019-12-12T20:55:34Z INFO	operator.go:199	operation 'operation-install' skipped for configurable.1.0

2019-12-12T20:55:36Z INFO	process.go:72	address assigned to the process '/home/travis/gopath/src/github.com/elastic/beats/x-pack/agent/pkg/agent/operation/tests/scripts/configurable-1.0-darwin-x86/configurable': '127.0.0.1:10000'

--- FAIL: TestConfigurableRun (2.51s)

    operator_test.go:209: found file /home/travis/gopath/src/github.com/elastic/beats/x-pack/agent/pkg/agent/operation/tests/scripts/configurable-1.0-darwin-x86/configurable

    operator_test.go:214: operation-start: failed to start '/home/travis/gopath/src/github.com/elastic/beats/x-pack/agent/pkg/agent/operation/tests/scripts/configurable-1.0-darwin-x86/configurable': fork/exec /home/travis/gopath/src/github.com/elastic/beats/x-pack/agent/pkg/agent/operation/tests/scripts/configurable-1.0-darwin-x86/configurable: operation not permitted

=== RUN   TestConfigurableByFileRun

2019-12-12T20:55:36Z INFO	operation_fetch.go:80	operation 'operation-fetch' downloaded configurablebyfile.1.0 into 

2019-12-12T20:55:36Z INFO	operator.go:199	operation 'operation-verify' skipped for configurablebyfile.1.0

2019-12-12T20:55:36Z INFO	operator.go:199	operation 'operation-install' skipped for configurablebyfile.1.0

--- FAIL: TestConfigurableByFileRun (0.00s)

    operator_test.go:294: found file /home/travis/gopath/src/github.com/elastic/beats/x-pack/agent/pkg/agent/operation/tests/scripts/configurablebyfile-1.0-darwin-x86/configurablebyfile

    operator_test.go:299: operation-start: chown /home/travis/gopath/src/github.com/elastic/beats/x-pack/agent/pkg/agent/operation/tests/scripts/configurablebyfile--1.0.yml: operation not permitted

michalpristas · 2019-12-16T10:39:48Z

seems like a permission problem. I've seen this happening in an incorrectly setup VM, root privileges are missing so you can chown or spawn a new process bin/sh

michalpristas

small nits. this includes this missing PR right?

ph · 2019-12-16T13:12:49Z

Weird, I can reproduce locally it in a freshly reset environment

michalpristas · 2019-12-16T13:15:54Z

check if you are the owner of a directory

michalpristas · 2019-12-16T14:26:02Z

@ph if you fix // +build directives as suggested build will pass

ph · 2019-12-16T14:54:07Z

@michalpristas Just pushed the commits for the previous comments, let's wait for the CI. 🤞

* Feature: Allow to persist AccessToken and other connection data on disk This PR add a few things - It takes the keystore encryption implementation and create an io.Reader and io.Writer for it, we currently work with variable lenght blocks and we are not compatible for now with the one in libbeat. - It implements a few storage mechanism and wrapper: - DiskStore: Save a io.reader directly to disk, the content is saved in a temporary file and the target is replace in an atomic operation. - EncryptedDiskStore: Same as Keystore but use the encrypted io.Reader and io.Writer. - A fleet configuration is created in _meta/agent.fleet.yml, the content of this file is packed in the binary. When a user enroll the agent, we do a backup of the current agent.yml and the content is replaced with the agent.fleet.yml - The enrollment information is saved into a "fleet.yml" - The managed mode now read the content of the fleet.yml and creates a Kibana API client. - A Separates Fleet Config struct is created. - Enroll will now ask for confirmation before replacing the user configuration. * allow to use the obfuscated store. * fix build

ph added 2 commits December 12, 2019 14:09

allow to use the obfuscated store.

6c2d9f7

ph added review [zube]: In Review Project:fleet labels Dec 12, 2019

ph requested a review from michalpristas December 12, 2019 20:28

ph self-assigned this Dec 12, 2019

ph mentioned this pull request Dec 12, 2019

Allow the Agent to retrieve configuration from the Fleet API. #15027

Merged

michalpristas approved these changes Dec 16, 2019

View reviewed changes

michalpristas reviewed Dec 16, 2019

View reviewed changes

Comment thread x-pack/agent/pkg/core/plugin/app/process_cred.go Outdated

michalpristas reviewed Dec 16, 2019

View reviewed changes

Comment thread x-pack/agent/pkg/core/plugin/app/process_cred_other.go Outdated

fix build

d2c1ab7

ph merged commit 9394639 into elastic:fleet Dec 16, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Store access token missing#15089

Store access token missing#15089
ph merged 3 commits intoelastic:fleetfrom
ph:store-access-token-missing

ph commented Dec 12, 2019

Uh oh!

elasticmachine commented Dec 12, 2019

Uh oh!

ph commented Dec 13, 2019

Uh oh!

michalpristas commented Dec 16, 2019 •

edited

Loading

Uh oh!

michalpristas left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ph commented Dec 16, 2019

Uh oh!

michalpristas commented Dec 16, 2019

Uh oh!

michalpristas commented Dec 16, 2019 •

edited

Loading

Uh oh!

ph commented Dec 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

ph commented Dec 12, 2019

Uh oh!

elasticmachine commented Dec 12, 2019

Uh oh!

ph commented Dec 13, 2019

Uh oh!

michalpristas commented Dec 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

michalpristas left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ph commented Dec 16, 2019

Uh oh!

michalpristas commented Dec 16, 2019

Uh oh!

michalpristas commented Dec 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ph commented Dec 16, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

michalpristas commented Dec 16, 2019 •

edited

Loading

michalpristas commented Dec 16, 2019 •

edited

Loading